Creation of representative content based on facial analysis

ABSTRACT

Technologies for analyzing various features detected in a face detected in an image. Such features may include at least the eyes, eyebrows, node, and mouth of a face. Such analyzing may include scoring aspects of these features. Such scores may be weighted. The analysis may be used to determine expressions and/or poses of the face, as well as indicate if the face is smiling, frowning, or neutral. An overall quality score for the face may also be determined and provided based on the aforementioned analysis as well as whether or not the face is near an edge of the image or cut-off in the image. Finally, a face signature may be determined that uniquely identifies the face, at least within the scope of the analyzed features. Such face signatures may be used to detect a similar face in other images.

BACKGROUND

Thanks to advances in imaging technologies, people take more picturesthan ever before. Further, the proliferation of media sharingapplications has increased the demand for picture sharing to a greaterdegree than ever before. Yet the flood of photos, and the need to sortthrough them to find relevant pictures, has actually increased the timeand effort required for sharing pictures. As a result, it is often thecase that either pictures that are less than representative of the bestpictures, or no pictures at all, end up getting shared.

SUMMARY

The summary provided in this section summarizes one or more partial orcomplete example embodiments of the invention in order to provide abasic high-level understanding to the reader. This summary is not anextensive description of the invention and it may not identify keyelements or aspects of the invention, or delineate the scope of theinvention. Its sole purpose is to present various aspects of theinvention in a simplified form as a prelude to the detailed descriptionprovided below.

The invention encompasses technologies for analyzing various featuresdetected in a face detected in an image. Such features may include atleast the eyes, eyebrows, nose, and mouth of a face as reflected in adetected image. Such analyzing may include scoring aspects of thesefeatures. Such scores may be weighted. The analysis may be used todetermine expressions and/or poses of the face, as well as indicate ifthe face is smiling, frowning, or neutral. An overall quality score forthe face may also be determined and provided based on the aforementionedanalysis as well as whether or not the face is near an edge of the imageor cut-off in the image. Finally, a face signature may be determinedthat uniquely identifies the face, at least within the scope of theanalyzed features. Such face signatures may be used to detect a similarface in other images. Reference to a face and to eyes, eyebrows, nose,mouth, and other features of the face as used herein typically refers tosuch as shown in an image as opposed an actual face or its features.

Many of the attendant features will be more readily appreciated as thesame become better understood by reference to the detailed descriptionprovided below in connection with the accompanying drawings.

DESCRIPTION OF THE DRAWINGS

The detailed description provided below will be better understood whenconsidered in connection with the accompanying drawings, where:

FIG. 1 is a block diagram showing an example computing environment inwhich the invention described herein may be implemented.

FIG. 2 is a block diagram showing an example system configured foranalyzing and scoring a face in an image.

FIG. 3 is a block diagram showing an example feature analyzer thatincludes sub modules such as face analyzer, eye analyzer, mouthanalyzer, and expression analyzer.

FIG. 4 is a diagram showing an example face with certain aspects of theface bounded by a rectangle that is associated with a face identifierthat uniquely identifies the face in the image.

FIG. 5 is a block diagram showing an example method for analyzing andscoring a face in an image.

FIG. 6 is a block diagram showing an example face analysis method.

FIG. 7 is a block diagram showing an example eye analysis method and anillustration of an example eye.

FIG. 8 is a block diagram showing an example mouth analysis method andillustrations of example mouths.

FIG. 9 is a block diagram showing an example expression analysis method.

Like-numbered labels in different figures are used to designate similaror identical elements or steps in the accompanying drawings.

DETAILED DESCRIPTION

The detailed description provided in this section, in connection withthe accompanying drawings, describes one or more partial or completeexample embodiments of the invention, but is not intended to describeall possible embodiments of the invention. This detailed descriptionsets forth various examples of at least some of the technologies,systems, and/or methods invention. However, the same or equivalenttechnologies, systems, and/or methods may be realized according toexamples as well.

Although the examples provided herein are described and illustrated asbeing implementable in a computing environment, the environmentdescribed is provided only as an example and not a limitation. As thoseskilled in the art will appreciate, the examples disclosed are suitablefor implementation in a wide variety of different computingenvironments.

FIG. 1 is a block diagram showing an example computing environment 100in which the invention described herein may be implemented. A suitablecomputing environment may be implemented with numerous general purposeor special purpose systems. Examples of well known systems include, butare not limited to, cell phones, personal digital assistants (“PDA”),personal computers (“PC”), hand-held or laptop devices,microprocessor-based systems, multiprocessor systems, systems on a chip(“SOC”), servers, Internet services, workstations, consumer electronicdevices, cell phones, set-top boxes, and the like. In all cases, suchsystems are strictly limited to articles of manufacture and the like.

Computing environment 100 typically includes a general-purpose computingsystem in the form of a computing device 101 coupled to variouscomponents, such as peripheral devices 102, 103, 101 and the like. Thesemay include components such as input devices 103, including voicerecognition technologies, touch pads, buttons, keyboards and/or pointingdevices, such as a mouse or trackball, that may operate via one or moreinput/output (“I/O”) interfaces 112. The components of computing device101 may include one or more processors (including central processingunits (“CPU”), graphics processing units (“GPU”), microprocessors(“μP”), and the like) 107, system memory 109, and a system bus 108 thattypically couples the various components. Processor(s) 107 typicallyprocesses or executes various computer-executable instructions and,based on those instructions, controls the operation of computing device101. This may include the computing device 101 communicating with otherelectronic and/or computing devices, systems or environments (not shown)via various communications technologies such as a network connection 114or the like. System bus 108 represents any number of bus structures,including a memory bus or memory controller, a peripheral bus, a serialbus, an accelerated graphics port, a processor or local bus using any ofa variety of bus architectures, and the like.

System memory 109 may include computer-readable media in the form ofvolatile memory, such as random access memory (“RAM”), and/ornon-volatile memory, such as read only memory (“ROM”) or flash memory(“FLASH”). A basic input/output system (“BIOS”) may be stored innon-volatile or the like. System memory 109 typically stores data,computer-executable instructions and/or program modules comprisingcomputer-executable instructions that are immediately accessible toand/or presently operated on by one or more of the processors 107.

Mass storage devices 104 and 110 may be coupled to computing device 101or incorporated into computing device 101 via coupling to the systembus. Such mass storage devices 104 and 110 may include non-volatile RAM,a magnetic disk drive which reads from and/or writes to a removable,non-volatile magnetic disk (e.g., a “floppy disk”) 105, and/or anoptical disk drive that reads from and/or writes to a non-volatileoptical disk such as a CD ROM, DVD ROM 106. Alternatively, a massstorage device, such as hard disk 110, may include non-removable storagemedium. Other mass storage devices may include memory cards, memorysticks, tape storage devices, and the like.

Any number of computer programs, files, data structures, and the likemay be stored in mass storage 110, other storage devices 104, 105, 106and system memory 109 (typically limited by available space) including,by way of example and not limitation, operating systems, applicationprograms, data files, directory structures, computer-executableinstructions, and the like.

Output components or devices, such as display device 102, may be coupledto computing device 101, typically via an interface such as a displayadapter 111. Output device 102 may be a liquid crystal display (“LCD”).Other example output devices may include printers, audio outputs, voiceoutputs, cathode ray tube (“CRT”) displays, tactile devices or othersensory output mechanisms, or the like. Output devices may enablecomputing device 101 to interact with human operators or other machines,systems, computing environments, or the like. A user may interface withcomputing environment 100 via any number of different I/O devices 103such as a touch pad, buttons, keyboard, mouse, joystick, game pad, dataport, and the like. These and other l/O devices may be coupled toprocessor 107 via I/O interfaces 112 which may be coupled to system bus108, and/or may be coupled by other interfaces and bus structures, suchas a parallel port, game port, universal serial bus (“USB”), fire wire,infrared (“IR”) port, and the like.

Computing device 101 may operate in a networked environment viacommunications connections to one or more remote computing devicesthrough one or more cellular networks, wireless networks, local areanetworks (“LAN”), wide area networks (“WAN”), storage area networks(“SAN”), the Internet, radio links, optical links and the like.Computing device 101 may be coupled to a network via network adapter 113or the like, or, alternatively, via a modem, digital subscriber line(“DSL”) link, integrated services digital network (“ISDN”) link,Internet link, wireless link, or the like.

Communications connection 114, such as a network connection, typicallyprovides a coupling to communications media, such as a network.Communications media typically provide computer-readable andcomputer-executable instructions, data structures, files, programmodules and other data using a modulated data signal, such as a carrierwave or other transport mechanism. The term “modulated data signal”typically means a signal that has one or more of its characteristics setor changed in such a manner as to encode information in the signal. Byway of example, and not limitation, communications media may includewired media, such as a wired network or direct-wired connection or thelike, and wireless media, such as acoustic, radio frequency, infrared,or other wireless communications mechanisms.

Power source 190, such as a battery or a power supply, typicallyprovides power for portions or all of computing environment 100. In thecase of the computing environment 100 being a mobile device or portabledevice or the like, power source 190 may be a battery. Alternatively, inthe case computing environment 100 is a desktop computer or server orthe like, power source 190 may be a power supply designed to connect toan alternating current (“AC”) source, such as via a wall outlet.

Some mobile devices may not include many of the components described inconnection with FIG. 1. For example, an electronic badge may becomprised of a coil of wire along with a simple processing unit 107 orthe like, the coil configured to act as power source 190 when inproximity to a card reader device or the like. Such a coil may also beconfigure to act as an antenna coupled to the processing unit 107 or thelike, the coil antenna capable of providing a form of communicationbetween the electronic badge and the card reader device. Suchcommunication may not involve networking, but may alternatively begeneral or special purpose communications via telemetry, point-to-point,RF, IR, audio, or other means. An electronic card may not includedisplay 102, I/O device 103, or many of the other components describedin connection with FIG. 1. Other mobile devices that may not includemany of the components described in connection with FIG. 1, by way ofexample and not limitation, include electronic bracelets, electronictags, implantable devices, and the like.

Those skilled in the art will realize that storage devices utilized toprovide computer-readable and computer-executable instructions and datacan be distributed over a network. For example, a remote computer orstorage device may store computer-readable and computer-executableinstructions in the form of software applications and data. A localcomputer may access the remote computer or storage device via thenetwork and download part or all of a software application or data andmay execute any computer-executable instructions. Alternatively, thelocal computer may download pieces of the software or data as needed, ordistributively process the software by executing some of theinstructions at the local computer and some at remote computers and/ordevices.

Those skilled in the art will also realize that, by utilizingconventional techniques, all or portions of the software'scomputer-executable instructions may be carried out by a dedicatedelectronic circuit such as a digital signal processor (“DSP”),programmable logic array (“PLA”), discrete circuits, and the like. Theterm “electronic apparatus” may include computing devices or consumerelectronic devices comprising any software, firmware or the like, orelectronic devices or circuits comprising no software, firmware or thelike.

The term “firmware” typically refers to executable instructions, code,data, applications, programs, program modules, or the like maintained inan electronic device such as a ROM. The term “software” generally refersto computer-executable instructions, code, data, applications, programs,program modules, or the like maintained in or on any form or type ofcomputer-readable media that is configured for storingcomputer-executable instructions or the like in a manner that isaccessible to a computing device. The term “computer-readable media” andthe like as used herein is strictly limited to one or more apparatus,article of manufacture, or the like that is not a signal or carrier waveper se. The term “computing device” as used in the claims refers to oneor more devices such as computing device 101 and encompasses clientdevices, mobile devices, one or more servers, network services such asan Internet service or corporate network service, and the like, and anycombination of such.

FIG. 2 is a block diagram showing an example system 200 configured foranalyzing and scoring a face in an image. The system includes severalmodules including facial recognition module 210 that accepts input 212,face detector module 214 that may be part of module 210, facial featuredetector module 216 that may be part of module 210, and feature analyzermodule 220 that accepts inputs 212, 215, and 217 and produces output222. Each of these modules (including any sub-modules) may beimplemented in hardware, firmware, software (e.g., program modulescomprising computer-executable instructions), or any combinationthereof. Each such module may be implemented on/by one device, such as acomputing device, or across multiple such devices. For example, onemodule may be implemented in a distributed fashion on/by multipledevices such as servers or elements of a network service or the like.Further, each such module (including any sub-modules) may encompass oneor more sub-modules or the like, and the modules may be implemented asseparate modules, or any two or more may be combined in whole or inpart. The division of modules (including any sub-modules) describedherein in non-limiting and intended primarily to aid in describingaspects of the invention. The phrase “face in the image” and the likerefers not to an actual face in an image, but to an image orrepresentation of a face, actual or otherwise.

In summary, system 200 is configured for analyzing and scoring a face inan image. An image(s) is typically provided by one or more sources asinput 212 to the system. Such sources include camera phones, digitalcameras, digital video recorders (“DVRs”), computers, digital photoalbums, social media applications, image and video streaming web sites,and any other source of digital images. Note that one or more actualimages may be input and/or output, or references to images, or anycombination of such.

Facial recognition module 210 is a module that accepts an image as input212, detects one or more faces in the image, and that detects variousfeatures in recognized faces. In one example, the functionality ofmodule 210 may be provided in the form of a software development kit(“SDK”). Module 210 may include sub modules such as face detector 214(e.g., the portion of module 210 that detects faces in an image) andfacial feature detector 216 (e.g., the portion of module 210 thatdetects facial features in an image). Any image provided at input 212 istypically available to both face detector 214 and facial featuredetector 216.

Face detector 214 typically provides output 215 in the form of a faceidentifier that identifies a detected face in image 212. Given multipledetected faces, a unique face identifier is typically provided for facedetected. In one example, a face identifier may be a RECT data structureor the like that bounds certain aspects of the face it identifies (e.g.,see example rectangle 410 in FIG. 4). Such a RECT data structure mayindicate a position in the image of the face it identifies, and/or mayindicate a size or relative size of the identified face in the image.Any face identifier(s) that are output 215 by face detector 214 may beaccepted as input by facial feature detector 216 and/or by featureanalyzer module 220 to uniquely identify a corresponding face(s) in theimage. Any suitable techniques may be used by face detector 214 todetect faces in input image 212.

Facial feature detector 216 typically accepts an image and a faceidentifier(s) of any face(s) detected in the image. Module 216 typicallyprovides output 217 in the form of a set of facial feature descriptorsthat describe facial features detected in a face corresponding to a faceidentifier. Given multiple face identifiers as input, a correspondingset of facial feature descriptors is typically provided for each faceidentifier. In one example, the set of facial feature descriptors may bein the form of coordinates for each detected facial features, such asdescribed in connection with FIG. 4. Any suitable techniques may be usedby facial feature detector 216 to detect facial features in faces ininput image 212.

Feature analyzer 220 typically accepts an image, a face identifier(s) ofany face(s) detected in the image, and a corresponding set of facialfeature descriptors for each face detected. Module 220 typicallyprovides output 222 in the form of face information for each facedetected. In one example, such face information may include, for eachface detected, the face identifier of the face, the face's set of facialfeature descriptors, a quality score of the face, a unique signature forthe face, an indication of an expression of the face, and indication ofa pose of the face, and/or additional analysis results such as describedherein below. Such indications may be provided in the form of a score orthe like, or in any other form. Such a face signature maybe used todetect, based at least on analyzed facial features, similar faces inother images, where such similar faces typically belong to the sameperson, closely-related persons, or similar-looking persons.

FIG. 3 is a block diagram showing an example feature analyzer 220 thatincludes sub modules such as face analyzer 310, eye analyzer 320, mouthanalyzer 330, and expression analyzer 340. Each such sub-module mayaccept as input an image, a face identifier for any face(s) detected inthe image, and a set of facial feature descriptors for each facedetected in the image. The following discusses each sub-module from theperspective of a single detected face in an image, but the functionalityof each sub-module may also apply to any number of faces detected in theimage.

Face analyzer 310 may analyze one or more aspects of a face. Forexample, module 310 may analyze the input to determine a location of theface in the image, a degree to which the face is cut-off in the image, adegree of sharpness of the face, and a confidence level that a detectedface actually represents a face. Examples of methods for the foregoingare provided below in connection with FIG. 6 below.

Eye analyzer 320 may analyze one or more aspects of an eye(s) in theface. For example, module 320 may analyze the input to determine adegree to which an eye is open, the direction an eye is looking, and/ora degree of sharpness of an eye in the image. Examples of methods forthe foregoing are provided below in connection with FIG. 7.

Mouth analyzer 330 may analyze one or more aspects of a mouth in theface. For example, module 330 may analyze the input to determine adegree to which the mouth is open, and a degree to which teeth arevisible in the mouth. Examples of methods for the foregoing are providedbelow in connection with FIG. 8.

Expression analyzer 340 may analyze one or more aspects of the face, theeyes, and the mouth in the face. For example, module 340 may analyze theinput to determine an expression on and/or a pose of the face. Examplesof methods for the foregoing are provided below in connection with FIG.9.

FIG. 4 is a diagram showing an example face with certain aspects of theface bounded by a rectangle 410 that is associated with a faceidentifier that uniquely identifies the face in the image. The exampleface in FIG. 4 is also marked with example points indicating coordinatesof various detected facial features of the face. Such coordinates may beidentified by a set of facial feature descriptors and may be relative toa reference point of rectangle 410, a reference point of the image thatincludes the face, or to some other reference point. Such faceidentifiers and sets of facial feature descriptors may be provided by amodule such as facial recognition module 210.

In one example, features on the face may be detected and indicated withcoordinate points such as a chin point 420 and face side points 421 and422. In addition, coordinate points 431, 432, and 433 may indicate adetected left eye brow, coordinate points 441, 442, and 443 mayindicated a detected right eye brow, and coordinate points 471 and 472may indicate a detected nose.

In the example, the left eye of the face may be detected and indicatedwith coordinate points 451-454, with coordinate point 450 indicating adirection the eye is looking. The right eye of the face may be detectedand indicated with coordinate points 461-464, with coordinate point 460indicating a direction the eye is looking. The phrases “direction theeye is looking”, “eye direction”, and the like generally refer to thedirection the eye is pointing or aimed. Further, the mouth of the facemay be detected and indicated with coordinate points 481-486.

FIG. 5 is a block diagram showing an example method 500 for analyzingand scoring a face in an image. Such a method may be performed byfeature analyzer 220 or the like. In one example, various inputs arereceived such as an image (at block 510), a face identifier thatidentifies a face detected in the image (at block 520), and a set offacial feature descriptors that describe facial features detected in aface corresponding to a face identifier (at block 530). One or more ofthe foregoing inputs are then typically used in analyzing and scoringvarious aspects of the face. The various inputs may be receivedseparately or in any combination. One or more of these inputs may beused in the various analysis steps performed in method 500. Further, inone example, at least steps 540, 550, 560, and 570 may be performed inany order as opposed to the order shown.

For example, block 540 of method 500 typically indicates analyzingaspects of the right eye of the face, and block 550 of method 500typically indicates analyzing aspects of the left eye of the face.Examples of eye analysis are provided below in connection with FIG. 7below. Block 560 of method 500 typically indicates analyzing aspects ofthe face. Examples of face analysis are provided below in connectionwith FIG. 6 below. Block 570 of method 500 typically indicates analyzingaspects of the mouth of the face. Examples of mouth analysis areprovided below in connection with FIG. 8 below. Block 580 of method 500typically indicates analyzing the expressions and the pose of the face.Examples of expression analysis are provided below in connection withFIG. 9 below.

Block 590 of method 500 typically indicates calculating a score for theface. Such a face score may be an overall quality score for the facethat is based on one or more of the analysis steps of method 500. In oneexample, the score may be a value between zero and one. In otherexamples, the score may be represented by a value within a continuousrange, or by a quantization such as high, medium, or low, or the like.In one example, the face score may represent an overall measure of thequality of the face in the image, and may be based on a combination ofanalyzed aspects such as face sharpness, face expression, face pose,and/or the state of the face's eyes and mouth. Such a score may beprovided as output 222 in response to image input 212.

FIG. 6 is a block diagram showing an example face analysis method 600.Such a method may be performed by face analyzer 310 or the like. In oneexample, method 600 includes confidence analysis 610, location analysis620, and sharpness analysis 630. In one example, various inputs tomethod 600 are provided, such as an image, a face identifier thatidentifies a face detected in the image, and a set of facial featuredescriptors that describe facial features detected in a facecorresponding to a face identifier. One or more of the foregoing inputsare then typically used in the face analysis.

Block 610 typically indicates presenting or determining a confidencelevel that a detected face represents an actual face. Such a level maybe received as part of the input, or may be determined based on datareceived as part of the input. In the latter case, the level may bebased on the face identifier and the set of facial feature descriptorsreceived as input. For example, a face indicator that indicates aproportionate face size relative to the image and/or a more complete andconsistent set of facial feature descriptors tend to indicate a higherconfidence, while an disproportionate face size and a less complete andconsistent set tend to indicate a lower confidence.

A confidence score may be assigned to the face that represents theconfidence level. Such a confidence score may be weighted, and may berepresented by a real number between zero and one, a value within someother range, and/or by a quantization such as high/medium/low, and/orthe like.

Block 620 typically indicates determining a location of the face in theimage. Based on this location, and the size of the face according to theface identifier, a proximity of the face to an edge of the image mayalso be determined. The proximity of the face to an edge of the imagemay be compared to an unacceptable proximity range resulting in alocation score for the face. Such a proximity range may be specified bya constant or by a variable that may be pre-set and/or definable by auser, a machine learning system, or otherwise. In one example, a lowerlocation score may be assigned if the face is located at least in partwithin the unacceptable proximity range than if the face is locatedentirely outside the range. Such a proximity range may be specified by aconstant or by a variable that may be pre-set and/or definable by auser, a machine learning system, or otherwise.

Such a face location score may be weighted, and may be represented by areal number between zero and one, by a value within some othercontinuous range, and/or by a quantization such as acceptable orunacceptable, or high, medium, or low, or the like. A user may be aperson or a system of any type.

Further, based on the face location and/or the facial featuredescriptors of the face, block 620 may include determining if the faceis cut-off at the edge of the image. The term “cut-off” as used hereingenerally refers to some portion of the face being located at an edge ofthe image such that a portion of the face is not shown in the image(i.e., the cut-off portion). A certain cut-off range may be defined(such as up to a certain percentage or the like of a face that iscut-off). Such a cut-off range may be specified by a constant or by avariable that may be pre-set and/or definable by a user, a machinelearning system, or otherwise.

A cut-off score may be assigned to the face that represents the degreeto which the face is cut off. Such a face cut-off score may be weighted,and may be represented by a real number between zero and one, by apercentage, by a value within some other range, and/or by a quantizationsuch as acceptable/unacceptable, cut off/not cut off, and/or the like.

Block 630 typically indicates determining a degree of sharpness of theface. In one example, the degree of sharpness may be based on pixelcontrast of lines of the face, where greater contrast across fewerpixels typically indicates a greater degree of face sharpness incontrast to lesser contrast across more pixels that typically indicatesa lesser degree of face sharpness.

A sharpness score may be assigned to the face that represents the degreeof sharpness. Such a face sharpness score may be weighted, and may berepresented by a real number between zero and one, by a value withinsome other range, and/or by a quantization such asacceptable/unacceptable, high/medium/low, and/or the like.

Block 640 typically indicates determining a face color. In one example,the color of the face may be determined from an area of the faceselected based on the facial features, such as areas proximate the eyesand/or nose. In one example, a area proximate the eyes and/or nose isselected and a dominant color in that area is determined. If thedetermined color corresponds to an expected hue, then the face color maybe assigned a higher score than if the determined color does notcorrespond to the expected hue. Such a face color score may indicate adegree of correspondence to the expected color. Further, a set ofexpected hues may be provided and used in score the determined facecolor.

Such a face color score may be weighted, and may be represented by areal number between zero and one, by a value within some other range,and/or by a quantization such as acceptable/unacceptable, estimatedethnicities based on the determined color, color descriptions (e.g.,black/white, etc), a closest corresponding member from the set ofexpected hues, and/or the like.

FIG. 7 is a block diagram showing an example eye analysis method 700 andan illustration of an example eye 740. Such a method may be performed byface analyzer 310 or the like. In one example, method 700 includes openanalysis 710, direction analysis 720, and sharpness analysis 730. In oneexample, various inputs to method 700 are provided, such as an image, aface identifier that identifies a face detected in the image, and a setof facial feature descriptors that describe facial features detected inthe face corresponding to a face identifier. One or more of theforegoing inputs are then typically used in the face analysis.

Block 710 typically indicates determining an open-closed state of an eyein the detected face. Such a state may be determined for the right eye,the left eye or both. The method may be performed in substantially thesame manner regardless of the particular eye being analyzed. In oneexample, the determination is made based on eye coordinate points, suchas points 461-464 shown for example eye 740. In this example, points 463and 464 typically indicate the broadest detected horizontal edges of theeye, and may be used to determine the eye's width as it appears in theimage. Points 461 and 462 typically indicate the broadest detectedvertical edges of the eye, and may be used to determine the eye's heightas it appears in the image. A ratio of the height to the width, or thewidth to the height, may indicate the open-closed state of the eye.

An eye open-closed score may be assigned to the eye and/or the face thatrepresents the eye's open-closed state. Such an eye open-closed scoremay be weighted, and may be represented by a real number between zeroand one, a ratio, a value within some other range, an opened and/orclosed percentage, and/or by a quantization such as open/closed, and/orthe like.

In some situations, such as when the face is turned to one side, onlyone of the horizontal edges of the eye may be detectable in the face. Insuch a situation, information that indicates a degree to which the faceis turned, along with the available eye coordinate points, may be usedto estimate the eye's width. Such information may be available as partof the input, and/or may be available as output from expression analyzer340 or from some other module. Such an estimated width may be used asthe determined eye's width in method 710.

Block 720 typically indicates determining a direction an eye is looking.Such a direction may be determined for the right eye, the left eye orboth. The method may be performed in substantially the same mannerregardless of the particular eye being analyzed. In order to determinethe direction an eye is looking, the open-closed state of the eye mayneed to be sufficiently open to enable detecting eye coordinate point460. The pose of the face may also need to be sufficient to enabledetecting eye coordinate point 460.

In one example, the determination is made based on eye coordinatepoints, such as points 460-464 shown for example eye 740. In thisexample, a location of point 460 in relation to points 462-463 and/or topoints 463-464 may be used in determining the direction the eye islooking. For example, if point 460 is located substantially equidistantfrom each of points 461-464, and if points 461 and 462 are locatedsubstantially equidistant from points 463 and 464, then the directionthe eye is looking may be determined to be substantially forward. Inthis example, forward may also be toward the camera that captured theimage.

Continuing the previous example, if point 460 is located to the right ofpoints 463-464, then the direction the eye is looking may be determinedto be to the right. If point 460 is located to the left of points463-464, then the direction the eye is looking may be determined to beto the left. If point 460 is located above points 463-464, then thedirection the eye is looking may be determined to be upward. If point460 is located below points 463-464, then the direction the eye islooking may be determined to be downward.

An eye direction score may be assigned to the eye and/or the face thatrepresents the eye direction. Such an eye direction score may beweighted, and may be represented by a real number between zero and one,an angle, a value in degrees, a value within some other range, and/or bya quantization such as toward the camera/not toward the camera, toward asubject of the image/not toward a subject of the image, forward/notforward, up/down, right/left, and/or the like.

Block 730 typically indicates determining a degree of sharpness of aneye in the detected face. Such a degree of sharpness may be determinedfor the right eye, the left eye or both. The method may be performed insubstantially the same manner regardless of the particular eye beinganalyzed. In one example, the degree of sharpness may be based on pixelcontrast of lines of the eye and/or around the eye, where greatercontrast across fewer pixels typically indicates a greater degree of eyesharpness in contrast to lesser contrast across more pixels thattypically indicates a lesser degree of eye sharpness.

A sharpness score may be assigned to the eye that represents the degreeof sharpness. Such an eye sharpness score may be weighted, and may berepresented by a real number between zero and one, by a value withinsome other continuous range, and/or by a quantization such as acceptableor unacceptable, or high, medium, or low, or the like.

FIG. 8 is a block diagram showing an example mouth analysis method 800and illustrations of example mouths 801, 802, and 803. Such a method maybe performed by mouth analyzer 330 or the like. In one example, method800 includes open analysis 810 and teeth analysis 820. In one example,various inputs to method 800 are provided, such as an image, a faceidentifier that identifies a face detected in the image, and a set offacial feature descriptors that describe facial features detected in theface corresponding to a face identifier. One or more of the foregoinginputs are then typically used in the mouth analysis.

Block 810 typically indicates determining an open-closed state of amouth in the detected face. In one example, the determination is madebased on mouth coordinate points, such as points 481-486 shown forexample mouths 801, 802, and 803. In this example, points 481 and 482typically indicate the broadest detected horizontal edges of the mouth,and may be used to determine the mouth's width as it appears in theimage. Points 483 and 484 typically indicate the top of the upper lipand the bottom of the upper lip respectively. Points 485 and 486typically indicate the top of the lower lip and the bottom of the lowerlip respectively. Points 483 and 486 typically indicate the broadestdetected vertical edges of the mouth, and may be used to determine themouth's height as it appears in the image. The mouth's height and widthmay be used to determine an area of the mouth on the face, or anapproximation or estimation thereof.

Further, points 484 and 485 may be used to determine a distance betweenthe bottom of the upper lip and the top of the lower lip, where such adistance may indicate the open-closed state of the mouth. A ratio of theheight to the distance, or the distance to the height, may indicate theopen-closed state of the mouth.

A mouth open-closed score may be assigned to the mouth and/or the facethat represents the mouth's open-closed state. Such a mouth open-closedscore may be weighted, and may be represented by a real number betweenzero and one, a ratio, a value within some other range, an opened and/orclosed percentage, and/or by a quantization such as open/closed, and/orthe like.

In some situations, such as when the face is turned to one side, onlyone of the horizontal edges of the mouth may be detectable in the face.In such a situation, information that indicates a degree to which themouth is turned, along with the available mouth coordinate points, maybe used to estimate the mouth's width. Such information may be availableas part of the input, and/or may be available as output from expressionanalyzer 340 or from some other module. Such an estimated width may beused as the determined mouth's width in method 810.

Block 820 typically indicates determining whether or not teeth arevisible in the mouth, or determining an amount of the mouth area inwhich teeth are visible. Recognition of teeth is typically based on thepresence of a pale color in an area of the mouth where teeth maytypically be anticipated. In one example, this area-termed herein the“teeth area”—is defined as the portion of an open mouth between points481 and 482 and between points 484 and 485, not including the lips ofthe mouth, or an approximation or estimation thereof.

If the open-closed state of the mouth is determined to be closed, thenit may also be determined that no teeth are visible, such as in example801. If the mouth is determined to be open, and a pale color is detectedin at least a portion of the teeth area of the mouth, then it may bedetermined that teeth are visible, such as in examples 802 and 803. Acomparison between the mouth area and the portion of the teeth area inwhich teeth are detected may be made to determine a measure of visibleteeth in the mouth.

A teeth visible score may be assigned to the mouth and/or face thatrepresents the determined measure of visible teeth. Such a teeth visiblescore may be weighted, and may be represented by a real number betweenzero and one, by a value within some other range, and/or by aquantization such as visible/not visible, and/or the like.

FIG. 9 is a block diagram showing an example expression analysis method900. Such a method may be performed by expression analyzer 330 or thelike. In one example, method 900 includes expression analysis 810 and/orpose analysis 820. In one example, various inputs to method 900 areprovided, such as an image, a face identifier that identifies a facedetected in the image, and a set of facial feature descriptors thatdescribe facial features detected in the face corresponding to a faceidentifier. One or more of the foregoing inputs are then typically usedin the expression and/or pose analysis.

Block 910 typically indicates determining an expression(s) of thedetected face. In one example, such determined expressions may beselected from a set comprising smiling, frowning, neutral, and other. Inthis example, determining such expressions may largely be based on themouth and mouth analysis 800. For example, the location of points 481and 482 relative to point 485 may be used to select one of theexpressions in the set. Smiling may be selected if points 481 and 482are located above point 485. Frowning may be selected if points 481 and482 are located below point 485. Neutral may be selected if points 481and 482 are located substantially in-line with point 485. Other may beselected for an arrangement of the points that is inconsistent with anyof those previously described.

An expression score may be assigned to the face that represents theface's expression. Such an expression score may be weighted, and may berepresented by a real number between zero and one, a value within someother range, and/or by a quantization such assmiling/frowning/neutral/other, and/or the like.

Block 920 typically indicates determining a pose(s) of the detectedface. Face poses may be determined based on information such as relativepositions of the eyes, nose, mouth, eyebrows, and the size of the face.Further, information describing or indicating the orientation of theface may be received or determined. All such information (receivedand/or determined) may be used to determine and indicate various posesof the face, such as forward or other facing, tilted up or down, rightor left, looking at the camera that captured the image, looking at asubject in the image, etc. One or more indications of determined and/orreceived poses may be provided. Further, a pose score may be assigned tothe face that represents the face's pose and/or pose quality. Such apose score may be weighted, and may be represented by a real numberbetween zero and one, a value within some other range, and/or by aquantization such as tilted right/left and/or up/down, and/or the like.

In view of the many possible embodiments to which the invention and theforgoing examples may be applied, it should be recognized that theexamples described herein are meant to be illustrative only and shouldnot be taken as limiting the scope of the present invention. Therefore,the invention as described herein contemplates all such embodiments asmay come within the scope of the following claims and any equivalentsthereto.

1. A method performed on a computing device, the method comprising:receiving information including an image, a face identifier thatidentifies a face detected in the image, and a set of facial featuredescriptors that indicate features of the face; analyzing, based on theset of descriptors, a mouth of the face and at least one eye of theface; and determining, based on the received information and theanalyzing, an expression of the face.
 2. The method of claim 1 where theanalyzing the at least one eye comprises determining an open-closedstate of the eye based on first points that indicate a width of the eyeand second points that indicate a height of the eye.
 3. The method ofclaim 1 where the analyzing the mouth determining an open-closed stateof the mouth based on first points that indicate a width of the mouthand second points that indicate a height of the mouth.
 4. The method ofclaim 1 further comprising determining an expression of the face basedon the received information and on the analyzing, where the determinedexpression is selected from a set comprising smiling, frowning, neutral,and other.
 5. The method of claim 4 where the determining comprisescomparing points that indicate horizontal edges of the mouth to at leastone point that indicates allocation of a lip of the mouth.
 6. The methodof claim 1 further comprising determining a face signature that uniquelyidentifies the face within a scope of analyzed features.
 7. The methodof claim 6 where the face signature is sufficient for identifying facesin other images that belong to a same person as does the face.
 8. Asystem comprising a computing device and at least one program modulethat are together configured for performing actions comprising:receiving information including an image, a face identifier thatidentifies a face detected in the image, and a set of facial featuredescriptors that indicate features of the face; analyzing, based on theset of descriptors, a mouth of the face and at least one eye of theface; and determining, based on the received information and theanalyzing, an expression of the face.
 9. The system of claim 8 where theanalyzing the at least one eye comprises determining an open-closedstate of the eye based on first points that indicate a width of the eyeand second points that indicate a height of the eye.
 10. The system ofclaim 8 where the analyzing the mouth determining an open-closed stateof the mouth based on first points that indicate a width of the mouthand second points that indicate a height of the mouth.
 11. The system ofclaim 8, the actions further comprising determining an expression of theface based on the received information and on the analyzing, where thedetermined expression is selected from a set comprising smiling,frowning, neutral, and other.
 12. The system of claim 11 where thedetermining comprises comparing points that indicate horizontal edges ofthe mouth to at least one point that indicates allocation of a lip ofthe mouth.
 13. The system of claim 8, the actions further comprisingdetermining a face signature that uniquely identifies the face within ascope of analyzed features.
 14. The system of claim 13 where the facesignature is sufficient for identifying faces in other images thatbelong to a same person as does the face.
 15. At least onecomputer-readable media storing computer-executable instructions that,when executed by a computing device, cause the computing device toperform actions comprising: receiving information including an image, aface identifier that identifies a face detected in the image, and a setof facial feature descriptors that indicate features of the face;analyzing, based on the set of descriptors, a mouth of the face and atleast one eye of the face; and determining, based on the receivedinformation and the analyzing, an expression of the face.
 16. The atleast one computer-readable media of claim 15 where the analyzing the atleast one eye comprises determining an open-closed state of the eyebased on first points that indicate a width of the eye and second pointsthat indicate a height of the eye.
 17. The at least onecomputer-readable media of claim 15 where the analyzing the mouthdetermining an open-closed state of the mouth based on first points thatindicate a width of the mouth and second points that indicate a heightof the mouth.
 18. The at least one computer-readable media of claim 15,the actions further comprising determining an expression of the facebased on the received information and on the analyzing, where thedetermined expression is selected from a set comprising smiling,frowning, neutral, and other.
 19. The at least one computer-readablemedia of claim 18 where the determining comprises comparing points thatindicate horizontal edges of the mouth to at least one point thatindicates allocation of a lip of the mouth.
 20. The at least onecomputer-readable media of claim 15, the actions further comprisingdetermining a face signature that uniquely identifies the face within ascope of analyzed features, where the face signature is sufficient foridentifying faces in other images that belong to a same person as doesthe face.